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ABSTRACT 

Over the past years, several Open Educational Resources (OERs) initiatives have been emerged worldwide aiming to 
create, share and reuse digital educational resources among educational communities. As a result, organizing, offering 
and accessing these resources over the web have been key issues for both the research and the educational community. 
Traditionally, a popular way for characterizing digital educational resources is by using a formal and centrally agreed 
classification system, such as the IEEE Learning Object Metadata (LOM). On the other hand, with the emerging Web 2.0 
applications, the issue of characterizing digital educational resources tends to move from the expert-based description 
based on formal classification systems to a less formal user-based tagging referred to as social tagging. As a result, a 
number of studies have been reported in the field of Technology-enhanced Learning (TeL) aiming to evaluate whether 
social tagging can improve the discovery of new digital educational resources and the retrieval of known ones stored in 
web-based repositories. Additionally, recent studies in the field of social tagging systems has provided initial evidence 
that users’ tagging motivation has a direct influence on the properties of the resulted tags and folksonomies. Thus, in this 
paper we aim to propose a methodology for investigating this issue and examine whether different end-users’ tagging 
motivations (that is teachers) could enlarge the metadata descriptions of digital educational resources. The results of our 
study provided us evidence that there is a direct influence of users' tagging motivation on the enlargement of metadata 
descriptions of digital educational resources, as well as to the resulted folksonomy when it is compared with formal 
structured vocabularies used by metadata experts or content providers for characterizing digital educational resources. 
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1. INTRODUCTION 

Over the past years, several Open Educational Resources (OERs) initiatives have been emerged worldwide 
towards the provision of open access to digital educational resources, in the form of Learning Objects (LOs) 
(McGreal, 2008). UNESCO (2002) has defined Open Educational Resources (OERs) as the “ technology- 
enabled , open provision of educational resources for consultation, use and adaptation by a community of 
users for non-commercial purposes ”, The main objective of OERs initiatives is to support the process of 
organizing, classifying and storing digital educational resources and their associated metadata in web-based 
repositories which are referred to as Learning Object Repositories (LORs) (McGreal, 2004). Within this 
context, a popular way for describing digital educational resources is by using a formal and centrally agreed 
classification system, such as the IEEE Learning Object Metadata (LOM) (IEEE LTSC, 2005). 

On the other hand, the emerging Web 2.0 applications have led to an enormous increase of the digital 
resources available on the web today. As a result, both the discovery of new resources and the retrieval of 
known ones on the web, become an increasingly complex problem (Heymann, Koutrika and Molina, 2008; 
Yande, Jatowt, Nakamura and Tanaka, 2007). Therefore, the issue of characterizing digital resources tends to 
move from the expert-based description based on formal classification systems to a less formal user-based 
tagging (that is, adding keywords to digital resources) (Derntl, Hampel, Motschnig-Pitrik and Pitner, 2011; 
Bi, Shang and Kao, 2009). Adding keywords, also known as tags, to any type of digital resource by users 
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(rather than resources’ authors) is referred to as Social Tagging (Bonino, 2009). The term of social tagging 
has emerged for those applications that encourage groups of individuals to openly share their private 
descriptions (or tags) of digital resources with other users, either by using a collection of tags created by the 
individuals for their personal use (referred to as folksonomy) (Anderson, 2007). 

In the field of Technology Enhanced Learning (TeL), social tagging has also been proposed as a mean for 
describing digital educational resources (Cho, Yeh, Cheng and Chang, 2011; Bateman, Brooks, McCalla and 
Brusilovsky, 2007). As a result, the generation of metadata is now done by individual users, who might 
primarily see private benefits, like an easy way to search and retrieve from LORs already used and known 
resources using meaningful terms (Dahl and Vossen, 2008). Thus, a number of studies have been reported 
aiming to evaluate the added value of socially tagging digital educational resources stored in LORs and 
compare it with the traditional approach of expert-based formal description based on centrally agreed 
classification systems, such as IEEE LOM (Trant, 2009b; Vuorikari and Ayre, 2009). Additionally, recent 
studies in the field of social tagging systems suggests that users’ tagging motivation has a direct influence on 
the properties of resulting tags and folksonomies (Korner, et al., 2010; Korner, 2009) but there are not 
existing studies for investigating this issue in the field of TeL. 

To this end, in this paper we aim to investigate this issue and we propose a methodology that aims to 
evaluate whether users’ tagging behaviour can influence (a) the enlargement of metadata descriptions of 
digital educational resources and (b) the resulted folksonomy when it is compared with formal structured 
vocabularies used by metadata experts or content providers for characterizing digital educational resources. 
The application of the proposed evaluation methodology in an existing LOR, namely the 
OpenScienceResources Repository (http://www.osrportal.eu/) provided us evidence that ‘verbose’ taggers 
can produce tags that are significantly different from formal metadata generated by metadata experts or 
content providers. 

The paper is structured as follows. Following this introduction. Section 2 provides an overview of 
existing studies from the literature, which have investigated the potential benefits of socially tagging digital 
educational resources. In Section 3, we present our proposed methodology for identifying different types of 
users’ tagging motivation and evaluating their possible influence to the metadata descriptions of digital 
educational resources, as well as to the resulted folksonomy when it is compared to formal structured 
vocabularies used for metadata authoring by metadata experts or content providers. Section 4, presents the 
tagging approaches adopted by the existing LOR (namely OpenScienceResources) where we applied our 
proposed methodology, the dataset of the OpenScienceResources repository that was available at the time of 
our study, as well as the evaluation results and discussion of our findings. Finally, we present our conclusions 
and ideas for future work. 


2. LITERATURE REVIEW 

Within the TeL literature there are existing works that have examined the added value of socially tagging 
digital educational resources. An initial study towards addressing this issue has been conducted in the 
framework of the EU-funded project “MELT: a Metadata Ecology for Learning and Teaching” 
(http://info.melt-project.eu/) (Zens et al., 2009; Vuorikari and Ayre, 2009). More specifically, MELT project 
developed a LOR including around 153.000 digital educational resources. These resources were 
characterized by the resources’ authors with educational metadata following an application profile of the 
IEEE LOM standard, as well as with social tags added by the end-users of the repository (that is students and 
teachers). During the evaluation study of the project, the MELT Repository included 10131 social tags, 
which had been added to 2008 digital educational resources. This means that on average 4 social tags were 
added per digital educational resource. The main issues that were investigated during the MELT project 
evaluation study were the following: 

■ II: usefulness of social tags for searching digital educational resources. 

■ 12 : usefulness of social tags as metadata descriptors of digital educational resources. 

The evaluation results in respect to the aforementioned issues showed that: 

■ II: 23% of searches in MELT Repository were performed based on social tags and 40% of these 
searches were found very useful. However, for accurate searches within MELT Repository, social 
tags were considered not useful. 
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■ 12 : 25% of social tags contained additional to authors’ information, 26% unnecessary information 
and 49% no new information. Moreover, social tags were found to be more useful than formal 
vocabulary terms and most of the end-users wanted to change the original metadata description to 
adopt some of the social tags as indexing terms. 

Another important study about social tagging has been conducted in the framework of the US-funded 
project “STEVE: The Museum Social Tagging Project” (http://www.steve.museum/) (Trant, 2009a). More 
specifically, the project developed a repository including around 97.000 digital resources of cultural heritage. 
These resources were characterized with metadata by professional museum experts as well as with tags added 
by the end-users of the repository (that is students and teachers). During the evaluation study of the project, 
the STEVE Repository was including 36981 tags which had been added to 1792 digital cultural heritage 
resources. This means that on average 20 social tags were added per cultural heritage resource. The main 
issues that were investigated during the STEVE project evaluation study were the following: 

■ II: social tags correlation with museum metadata. 

■ 12 : usefulness of social tags as museum metadata descriptors of the digital cultural heritage 
resources. 

The evaluation results in respect to the aforementioned issues showed that: 

■ II: 86% of social tags didn’t match with museum metadata added by professional museum experts. 

■ 12 : 88% of social tags were considered useful by the professional museum experts 

As we can notice from the aforementioned studies, there is a strong interest for investigating the added 
value of social tagging on enlarging the metadata descriptions of digital educational resources, as well as the 
formal vocabularies used in expert-based metadata. However, both issues have been investigated without 
considering the possible implications of users’ tagging motivation to the enlargement of resulting tags and 
folksonomies. Next, we aim to address this issue and we present our proposed evaluation methodology. 


3. PROPOSED EVALUATION METHODOLOGY 

In this section, we present our proposed methodology for identifying different types of users’ tagging 
motivation and evaluating their possible influence to the metadata descriptions of digital educational 
resources, as well as to the resulted folksonomy when it is compared to formal structured vocabularies used 
for metadata authoring by metadata experts or content providers. More specifically, our methodology 
includes the following steps: 

■ Step 1 - Identify different underlying behaviours for users’ tagging: This step includes the 
discrimination of the users based on their tagging motivation. For this purpose, we adopt two types 
of user motivations proposed by Komer (2009): 

o Categorizers, who are motivated by categorization and use tags to construct and maintain a 
navigational aid to the resources they annotate. For this purpose, categorizers aim to 
establish a stable and bounded vocabulary based on their personal preferences and 
motivation. 

o Describers, who are motivated by description aim to describe the resources they annotate 
accurately and precisely. As a result, their tag vocabulary typically contains an open set of 
tags. 

In order to discriminate between categorizers and describers we adopt a set of measures proposed by 
Korner et al. (2010): 

o Tag/Resource Ratio : relates the vocabulary size of a user to the total number of digital 
educational Resources tagged by this user. Describers, who use a variety of different tags 
for their resources, score higher values for this measure than categorizers, who use fewer 
tags. 

o Orphaned Tag Ratio: characterizes the degree to which users produce orphaned tags (that is 
tags assigned to few resources only, and therefore are used infrequently). The orphaned tag 
ratio captures the percentage of tags in a user's vocabulary that represent such orphaned 
tags. Categorizers vocabulary scores values closer to 0 because orphaned tags would 
introduce noise to their personal vocabulary, whereas describers vocabulary scores values 
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closer to 1 due to the fact that describers tag resources in a more verbose and descriptive 
way. 

o Overlap Factor: measures the phenomenon of an overlap produced by the assignment of 
more than one tag per resource on average. Categorizers are interested in keeping this 
overlap relatively low. On the other hand, describers produce a high overlap factor since 
they do not use tags for navigation but instead aim to best support later retrieval. 
o Tag/Title Intersection Ratio: measures how likely users choose tags from the words of an 
educational resource title. Categorizers use tags taken from the title and they score values 
closer to 1, whereas describers rarely use tags from the title and they score values closer to 
0 . 

■ Step 2 - Calculate similarity between social tags and educational metadata: During this step we 
calculate the similarity between social tags (offered by end-users, that is teachers) and educational 
metadata (offered by metadata experts or content providers). The similarity is calculated for social 
tags added by describers, as well as for social tags added by categorizers based on the users’ 
discrimination performed in step 1 . At the end of this step, we expect to identify digital educational 
resources enlarged with social tags offered by describers and/or categorizers that are different by the 
formal metadata descriptions offered by metadata experts or content providers. 

■ Step 3 - Compare folksonomy with formal vocabularies of educational metadata: During this 
step, we compare the resulted folksonomy produced by the social tags with formal structured 
vocabularies of educational metadata. The comparison is performed with the folksonomy produced 
by describers, as well as with the folksonomy produced by categorizers following the users’ 
discrimination performed in step 1. At the end of this step, we would be able to identify new tags 
offered by describers and/or categorizers that can enlarge the formal structured vocabularies of 
educational metadata. 


4. APPLYING PROPOSED METHODOLOGY TO AN EXISTING 
LEARNING OBJECT REPOSITORY 

In this section, we apply our proposed evaluation methodology to an existing LOR, namely 
OpenScienceResources Repository (http://www.osrportal.eu/). The OpenScienceResources Repository was 
developed in the framework of an EU-funded project, referred to as “OpenScienceResources: Towards the 
development of a Shared Digital Repository for Formal and Informal Science Education” 
(http://www.openscienceresources.eu/). It provides access to openly licensed (through Creative Commons) 
science education resources, which can be used by science teachers connecting formal science education in 
schools with informal science education activities taken place in European Science Centres and Museums 
(Sampson, Zervas and Sotiriou, 2011). 

The OpenScienceResources Repository currently includes around 3000 science education resources 
characterized by content providers (European Science Centres and Museums) with educational metadata 
following an application profile of the IEEE LOM standard (Sampson, Zervas and Sotiriou, 2011). More 
specifically, the application profile of the OpenScienceResources Repository includes formal structured 
vocabularies at the Classification Element (Nr. 9) of the IEEE LOM standard for describing (a) the subject 
domain of science education resources that is related with the science curriculum (the vocabulary includes 
400 terms) and (b) the educational objectives that science education resources intend to target (the 
vocabulary includes terms selected by Bloom’s Taxonomy of Educational Objectives and its subsequent 
revisions and extensions (Anderson and Krathwohl, 2001)). 

The science education resources of the OpenScienceResources Repository are also characterized with 
social tags added by the end-users of the repository (that is teachers). The social tags includes the following 
categories: (a) free tags, that are mainly used by the end-users, so as to describe the topic and/or the subject 
domain of a science education resource related with the science curriculum and (b) educational objectives 
tags, that are used by the end-users, so as to express the educational objective(s) that a science education 
resource can target. The educational objectives tags are selected from a predefined vocabulary following 
revised Bloom’s Taxonomy of Educational Objectives and it is the same with the formal vocabulary, used by 
content providers for characterizing science education resources with IEEE LOM compliant educational 
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metadata. Table 1 presents the two tagging approaches of the OpenScienceResources Repository and how the 
metadata elements added by content providers can be compared with the social tags added by end-users. 

Table 1. OpenScienceResources Repository Tagging Approaches 


Educational Metadata (according to IEEE LOM) 


Metadata Element 
1 .5 General. Key word 

9. 2. 2. 2 Classification.Taxon 
Path.Taxon.Entry (when purpose 
metadata element has the value 
“discipline”) 

9. 2. 2. 2 Classification.Taxon 
Path.Taxon.Entry (when purpose 
metadata element has the value 
“educational objective”) 


Value Space 

Free text describing the topic of 
the science education resource 
Structured vocabulary describing 
the subject domain of the science 
education resource related with 
the science curriculum 

Structured vocabulary describing 
the educational objectives that a 
science education resource intends 
to target 


Social Tags 


Tags Categories 
Free Tags 


Educational 
Objectives Tags 


Value Space 
Free text describing 
the topic and/or the 
subject domain of a 
science education 
resource related with 
the science 
curriculum 

Structured vocabulary 
describing the 
educational objectives 
that a science 
education resource 
intends to target 


Next, we present the snapshot of the OpenScienceResources Repository dataset at the time of our study 
(March 2012) that was used for applying our proposed evaluation methodology described in Section 3. 

Table 2. OpenScienceResources Dataset (March 2012) 


Variables 

Value 

Taggers 

434 

Tagged Science Education Resources 

1877 

Social Tags 

14707 

Free Tags 

13117 

Educational Objectives Tags 

1590 


4.1 Describers and Categorizers 

In order to cluster the taggers of the OpenScienceResources Repository we applied the set of measures 
described in our proposed methodology in Section 3. Table 3 presents the number of categorizers and 
describers resulted by applying these measures. 

Table 3. Describers and Categorizers 


Type of Taggers 

Value 

% per Total Taggers 

Categorizers 

226 

52,07 % 

Describers 

208 

47,92 % 

Total 

434 

100,00 % 


Additionally, Table 4 presents quantitative data about the number of tags used and the number of science 
education resources tagged by each type of tagger. 


Table 4. Quantitative Data per Type of Tagger 


Type of 

Number 

Tags 

Average Tags 

Science Education 

Average Tagged 

Average Tags 

Taggers 

of 

Taggers 

Contributed 

per Tagger 

Resources Tagged 

Science Education 
Resources per 
Tagger 

per Science 
Education 
Resources 

Categorizers 226 

1960 

8,67 

1852 

8.19 

1,05 

Describers 

208 

12647 

60,8 

630 

3,02 

20,07 


As it was expected we can notice from Table 4 that describers contributed the vast majority of social tags 
of the OpenScienceResources dataset but they have tagged less science education resources than the 
categorizers. This means that describers mainly use a small set of digital educational resources, which they 
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want to accurately and precisely describe for later searching and retrieval. On the other hand categorizers 
contributed a small amount of social tags to a large set of digital resources aiming to support future browsing 
to as many educational resources of the repository. 

4.2 Educational Metadata vs. Social Tags 

The next step of our proposed evaluation methodology includes similarity calculation between educational 
metadata and social tags. More specifically, we calculated (a) the similarity of the metadata values added by 
content providers to the element Nr. 1 .6 Keyword of the IEEE LOM standard with the free tags category of 
the social tags added by describers and categorizers and (b) the similarity of the metadata values added by 
content providers to the element Nr. 9.2.2.2 Classification. TaxonPath. Taxon. Entry (when purpose metadata 
element has the value “educational objective”) of the IEEE LOM standard with the educational objectives 
tags category of the social tags added by describers and categorizers. Table 5 presents average similarity 
results for the different type of taggers, as well as the number of science education resources that achieved 
low similarity score in average (<0,5). This similarity threshold was selected by considering relevant 
thresholds used in other social tagging evaluation studies from the literature (Vuorikari and Ayre, 2009). 


Table 5. Educational Metadata vs. Social Tags 


Type of Tagger: Keyword vs. 

Classification.Taxon 

Science Education 

% per total tagged 

Free Tags 

Path.Taxon.Entry vs. 

Resources with low 

science education 


Educational Objectives 

similarity score (<0,5) 

resources 


Tags 



Categorizers 0,79 

0,71 

184 out of 1852 

9,93% 

Describers 0,76 

0,45 

307 out of 630 

48,73% 


As we can notice from Table 5, describers added social tags (mainly to the educational objectives tags 
category) that were significantly different from the educational metadata added by the content providers. 
Therefore, they have significantly contributed to the enlargement of the metadata descriptions of 307 science 
education resources. This was the 48,37% of the science education resources that they have tagged. On the 
other hand, categorizers added tags that were quite similar with the educational metadata added by the 
content providers. Therefore, this distinction between categorizers and describers is very important because it 
facilitates capturing enlarged metadata descriptions of digital educational resources. 

4.3 Folksonomy vs. Formal Structured Vocabulary 

During the final step of our proposed evaluation methodology, we compared the resulted folksonomy of free 
tags category of social tags added by describers and categorizers with the structured formal vocabulary used 
by content providers to characterize science education resources with terms related with science curriculum. 
In order to achieve that, we calculated the similarity of the describers folksonomy and categorizers 
folksonomy with the formal structured vocabulary and we kept those tags that they achieved similarity score 
zero. The number of these tags was 202 (from describers) and 94 (from categorizers). Finally, we excluded 
the semantic noise from these tags, that is synonyms and subjective tags and we concluded to 46 (describers) 
and 10 (categorizers) possible new terms that could enlarge the structured formal vocabulary used at the 
element Nr. 9.2.2. 2 Classification. TaxonPath. Taxon. Entry (when purpose metadata element has the value 
“discipline”) of the IEEE LOM standard. Finally, we investigated whether there were new terms contributed 
by both describers and categorizers. Table 6 presents the distribution of the new terms contributed. 


Table 6. Distribution of new terms per type of tagger 


Type of Taggers 

New terms Contributed 

% per Total Terms 

Only by Describers 

46 

82,14 % 

Only by Categorizers 

10 

17,85 % 

By Categorizers & Describers 

0 

0,00 % 


As we can notice from Table 6, describers outperformed categorizers by contributing 36 more new terms. 
Moreover, there were not common terms contributed by both describers and categorizers. These results 
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provided us evidence that describers have a stronger influence on the enlargement of formal structured 
vocabularies, whereas categorizers’ contribution was limited. These findings could be relevant in LORs for 
indexing digital educational resources with describers’ tags towards facilitating search and retrieval of digital 
educational resources. 


5. CONCLUSIONS AND FUTURE WORK 

In this paper, we investigated the influence of different tagging styles, namely describers and categorizers on 
enlarging the metadata descriptions of digital educational resources. The results of our study performed with 
the dataset of an existing LOR, namely OpenScienceResources Repository showed that describers produce 
tags that are significantly different from formal metadata, whereas categorizers mainly follow the formal 
metadata generated by metadata experts or content providers. As a result, considering tagging motivation 
during folksonomies analysis could facilitate capturing enlarged metadata descriptions of digital educational 
resources. Future work includes deeper analysis to the results of our study, so as to identify the effect of 
describers’ and categorizers’ social tags to the enlargement of metadata descriptions for digital educational 
resources with different granularity levels and different formats. 
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